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PREFACE 


In  this  briefing  note,  the  authors  present  the  results  of  an 
investigation  of  the  applicability  of  the  subjective  transfer  function 
approach  to  complex  system  analysis--as  developed  by  Rand[l)--to  Air- 
Force-Wide  Mission  Area  Analysis.  The  work  was  requested  by  the 
Capability  Assessment  Division,  Directorate  of  Plans,  Office,  DCS/Plans 
and  Operations,  Headquarters,  United  States  Air  Force.  It  was  conducted 
as  part  of  the  Concept  Development  and  Project  Formulation  project  of 
the  Project  AIR  FORCE  Resource  Management  Program. 


[1]  See  C.  Veit  and  M.  Callero,  Subjective  Transfer  Function  Ap¬ 
proach  to  Complex  System  Analysis ,  The  Rand  Corporation,  R-2719-AF, 
February  1981. 


SUMMARY 


This  Note  describes  an  application  of  the  subjective  transfer 
function  (STF)  approach  to  complex  system  analysis  (Veit  and  Callero, 
1981)  to  an  Air-Force-Wide  Mission  Area  Analysis  (AFWMAA)  problem 
domain.  The  application  demonstrates  that  the  STF  approach  can  be 
readily  used  by  AFWMAA  to  establish  a  basis  for  evaluating  Program 
Decision  Packages  and  for  generating  subjective  inputs  to  utility  models 
supporting  major  mission  areas.  Furthermore,  the  STF  approach  has  the 
advantages  of  (1)  clearly  and  efficiently  representing  the  problem 
domain,  (2)  empirically  testing  the  representation  and  the  relationships 
among  its  factors  that  explain  how  they  produce  outcomes  of  interest, 

(3)  illuminating  and  accounting  for  synergistic  effects  among  factors, 
and  (4)  directly  showing  tradeoffs  among  the  factors  for  guiding 
decisionmaking. 

The  selected  AFWMAA  problem  domain  was  immediate  targeting  for 
battlefield  air  interdiction  (BAI)  in  the  tactical  command  and  control 
mission  area.  The  initial  representation  of  the  problem  domain 
consisted  of  five  sets  of  factors  hypothesized  to  either  directly,  or 
indirectly  through  outcomes  internal  to  the  structure,  affect  the 
ability  to  do  BAI  immediate  targeting.  Respondents  answering 
questionnaires  fielded  for  study  were  Air  Force  officers  assigned  to  the 
Pacific  Air  Forces  (PACAF)  and  Headquarters,  Pacific  Command. 

Results  of  data  analyses  indicated  that  the  initial  representation 
should  be  altered  to  consist  of  four  (instead  of  the  originally 
hypothesized  five)  factor  sets,  with  a  separate  STF  to  link  each  set  of 
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factors  to  its  outcome.  Three  STFs  were  selected  to  account  for 
interactions  observed  in  the  data  among  many  of  the  factors.  Two  kinds 
of  interactions  were  observed.  For  two  sets,  when  capability  was  poor 
for  one  factor,  another  factor  affected  judgments  less ;  e.g.,  the 
ability  of  the  tactical  air  control  facilities  to  support  the 
immediate-targeting  function  made  less  difference  when  the  ability  to 
communicate  tasking  orders  to  the  tactical  air  forces  was  poor  than  when 
it  was  good.  For  another  set,  when  capability  was  poor  for  one  factor, 
another  factor  had  a  greater  effect  on  judgments;  e.g.,  the  capability 
of  gathering  enemy  emitter  data  made  a  greater  difference  to  respondents 
when  capability  for  gathering  enemy  vehicle  data  was  poor  than  when  it 
was  good.  A  fourth  STF  best  explained  the  primarily  additive 
(noninteractive)  data  found  in  the  fourth  set. 

The  resulting  structure  and  STFs  provide  the  basis  for  evaluating 
command  and  control  and  intelligence  collection  system  configurations  in 
terms  of  how  well  the  BAI  immediate  targeting  function  could  be 
performed.  Examples  are  provided  using  notional  configurations  to 
demonstrate  how  Program  Decision  Packages  would  be  evaluated  and  inputs 
provided  to  theater  models. 
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DEMONSTRATION  OF  SUBJECTIVE 
TRANSFER  FUNCTION  APPROACH 
APPLIED  TO  AFWMAA 


Slide  1 

This  briefing  note  reports  on  the  application  of  the  subjective 
transfer  function  (STF)  approach  to  an  Air-Force-Wide  Mission  Area 
Analysis  (AFWMAA)  problem  domain. 
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OUTLINE 


•  Background 

•  STF  approach  overview 

•  Application 

•  Significant  features 
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Above  is  an  outline  of  the  note.  Following  a  short  description  of 
the  background  that  led  up  to  the  research  and  an  overview  of  the  STF 
approach,  we  will  discuss  how  the  STF  approach  was  applied  to  a  selected 
AFWMAA  problem.  That  section  includes: 

o  the  hypothesized  structure  of  the  AFWMAA  problem  domain; 

o  how  the  judgment  data  were  gathered; 

o  the  judgment  models  entertained  as  STFs  to  explain  how  the 

components  comprising  the  structure  affect  judged  outcomes,  and 

o  how  the  data  were  analyzed  to  provide  information  about  the 
appropriateness  of  the  hypothesized  structure  and  the 
appropriate  STFs  for  functionally  linking  the  components  of  the 
structure ; 


o  how  resulting  STFs  would  be  used  to  assess  the  capabilities  of 
alternate  command  and  control  systems. 


Finally,  we  describe  the  significant  features  of  the  STF  approach  and 
briefly  summarize  the  paper. 
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BACKGROUND 


•  Initiated  by  XOXIM 

•  Investigate  improved  techniques 

•  Demonstration  effort 

•  Tactical  C2/BAI/immediate  targeting 

-  PDP  assessment 

-  Theater  model  support 
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The  study  was  initiated  by  the  Capability  Assessment  Division 
(XOXIM),  Directorate  of  Plans,  Office,  DCS/Plans  and  Operations , 
Headquarters,  United  States  Air  Force,  in  order  to  investigate  improved 
methods  for  conducting  AFWMAA. 

It  was  decided  that  Rand's  effort  would  be  to  demonstrate  the  STF 
approach  to  a  representative  MAA  problem  domain.  The  domain  selected 
was  in  the  tactical  command  and  control  portion  of  the  orientation 
mission  area.  Specifically,  it  was  the  immediate  targeting  function 
within  the  battlefield  air  interdiction  (BAI)  mission  area.  We  were 
particularly  interested  in  demonstrating  how  program  decision  packages 
(PDPs)  would  be  assessed  and  how  inputs  to  the  MAA  theater  model  would 
be  generated  using  the  STF  approach. 
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STF APPROACH 


Input  factors 


Factor  levels 
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This  slide  is  a  generic  representation  of  a  small  problem  domain-- 
one  set  of  input  factors  linked  by  a  STF  (T(.\,Y,Z))  to  a  single  outcome. 
Three  aspects  of  the  STF  approach  will  be  pointed  out  using  this  generic 
representation. 

1.  The  problem  domain  is  causally  structured;  it  is  comprised  of 
sets  of  proposed  causally  linked  factors  (one  set  of  causally  linked 
factors  is  shown  in  this  slide).  This  set  depicts  the  hypothesis  that 
input  factors,  x,  y,  and  z,  impact  on  the  outcome  factor.  Each  input 
factor  is  described  in  terms  of  its  factor  levels.  For  example,  if  the 
outcome  of  interest  pertained  to  making  immediate  targeting  decisions, 
factor  x  might  be  the  currency  of  enemy  information  coming  into  the 
command  and  control  system.  The  currency  factor  levels  might  be  one 
hour,  30  minutes,  15  minutes,  and  five  minutes.  Factor  y  might  be  the 
percentage  of  the  BAI  area  that  is  covered,  with  factor  levels  ranging 
from  90%  to  10%.  Factor  z  would  be  another  hypothesized  input  factor. 


The  idea  in  the  STF  approach  is  to  determine,  from  experts'  judgments  of 
the  outcomes,  the  subjective  transfer  function  (T(x,y,z))  that  links  the 
input  factors  (described  by  their  factor  levels)  and  the  outcome  they 
affect.  The  T  is  the  algebraic  model  that  specifies  how  the  experts' 
subjective  values  of  the  input  factor  levels  affect  their  subjective 
outcome  values. 

2.  In  the  STF  approach,  transfer  functions  (Ts)  and  the  proposed 
causal  structure  are  simultaneously  verified.  To  do  this, 
questionnaires  are  generated  from  factorial  combinations  of  the  input 
factor  levels  and  given  to  informed  respondents.  Each  item  on  the 
questionnaire  is  comprised  of  a  different  combination  of  factor  levels 
and  thus  represents  a  different  set  of  command  and  control  capabilities. 
To  each  item,  the  respondent  judges  what  the  outcome  (along  the 
specified  judgment  dimension)  would  be  in  that  situation.  When 
statistical  analyses  of  respondents'  data  indicate  that  one  or  more  of 
the  proposed  input  factors  does  not  affect  their  judged  outcomes,  these 
factors  are  eliminated  from  the  structure.  If  respondents  find  the  task 
difficult  because  important  information  is  missing  or  if  much  of  their 
data  is  internally  inconsistent,  other  factors  are  sought  (usually 
through  interaction  with  the  respondents)  to  describe  the  structure. 

The  design  that  generated  the  questionnaire  makes  it  possible  to  test 
different  algebraic  formulations  of  T.  Only  after  appropriate  Ts  are 
found  to  explain  all  the  judgment  data,  does  the  final  causal  structure 
emerge . 

3.  Once  an  appropriate  T  is  found,  it  is  possible  to  use  it  to 
predict  what  the  outcome  would  be  for  each  possible  combination  of  the 


input  factor  levels  and  hence  each  different  set  of  command  and  control 
capabilities . [ 1 ] 


[l]When  input  factor  levels  are  measured  using  physical  values 
(e.g.,  minutes,  meters,  percent),  the  function  relating  those  physical 
values  to  their  subjective  counterparts  (referred  to  as  the  psychophysi 
cal  function  or  utility  curve)  is  derived  from  the  transfer  function,  T 
Thus  for  those  factors,  the  appropriate  subjective  value  can  be  interpo 
lated  from  the  function  for  any  physical  value. 
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STF  APPROACH 


Slide  5 


This  slide  depicts  a  complex  problem  domain  consisting  of  four 
causal  sets  and  four  subjective  transfer  functions  (Ts).  Verification 
of  the  Ts  serve  to  validate  the  structure  (the  factors  and  their 
hierarchical  arrangement)  and  functionally  interlink  the  sets.  When  the 
STFs  are  known,  it  is  possible  to  assess  how  changes  in  input  factor 
levels  at  the  bottom  of  the  structure  affect  outcomes  throughout  the 
structure  and  at  the  top. 
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STF  APPLICATION 

•  Structure 

•  Data-gathering  methods 

•  Tests  of  proposed  transfer  functions 

•  Transfer  functions 

•  Outcomes 
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The  topics  to  be  covered  in  describing  the  application  of  the  STF 
approach  to  a  MAA  problem  domain  are  outlined  above.  We  will  describe 
the  hypothesized  factors  and  their  causally  linked  sets  that  make  up  the 
structure;  data-gathering  methods;  tests  of  transfer  functions  to 
explain  the  judgment  data;  the  actual  transfer  functions  that  were 
supported  by  the  respondents'  data;  and  some  outcomes  implied  by  those 
functions . 
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HYPOTHESIZED  IMMEDIATE  TARGETING  STRUCTURE 


Slide  7 


The  problem  domain  structure  used  in  our  demonstration  study  was 
that  of  immediate  targeting  in  the  battlefield  air  interdiction  (BAI) 
area.  This  slide  outlines  the  factors  and  factor  structure  initially 
hypothesized  to  depict  that  problem  domain.  The  domain  is 
hierarchically  structured.  At  the  top,  three  factors  are  hypothesized 
to  impact  on  BAI  operations.  These  are  the  ability  to  plan  those 
operations  (Planning),  the  ability  to  do  immediate  targeting  (Immediate 
Targeting),  and  the  status  of  weapons  control  (Weapons  Control). 

Because  this  study  focused  on  immediate  targeting,  the  remaining 
structure  only  depicts  the  factors  hypothesized  to  affect  that 
component . 

Factors  hypothesized  to  directly  impact  on  immediate  targeting  are 


information  about  the  enemy  (Enemy  Second  Echelon  Force  Information); 
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information  about  the  relevant  friendly  forces  and  the  weather 
(Execution  Status  Information);  the  adequacy  of  the  facilities  within 
which  the  immediate-targeting  function  takes  place  (Facility 
Operability);  and  the  ability  to  disseminate  immediate-targeting 
decisions  to  the  forces  (Dissemination). 

Three  factors--information  about  alert  forces  (Alert  Forces), 
information  about  airborne  forces  (Airborne  Forces),  and  information 
about  the  weather  (Weather) --are  hypothesized  to  impact  on  immediate 
targeting  through  Execution  Status  Information  (T(2)). 

TI.ree  factors  are  hypothesized  to  impact  on  enemy  second  echelon 
force  information  (T(3)).  These  are:  enemy  vehicle  information 
(Vehicles);  enemy  radio/electronic  emitter  information  (Emitters);  and 
the  means  by  which  that  information  is  processed  (Processing). 

Three  factors  are  hypothesized  to  impact  on  enemy  vehicle 
information  (T(4)):  how  well  the  enemy  vehicles  can  be  located  and 
classified  (Location/  Classification);  how  many  of  them  can  be  observed 
(Coverage) ,  and  how  long  it  takes  to  report  vehicle  observations  to  the 
command  and  control  system  for  processing  (Currency).  Three  similar 
factors.  Location,  Coverage,  and  Currency,  are  hypothesized  to  impact  on 
enemy  emitter  information  (T(5)).  As  can  be  seen,  all  of  the  factors  at 
the  bottom  of  the  hierarchy  impact  on  immediate  targeting  at  the  top 
through  their  subjective  transfer  function  links. 

The  factors  comprising  the  immediate-targeting  structure  were 
separated  into  two  sections  that  correspond  to  the  two  questionnaires 
fielded  for  study.  Factors  used  to  generate  the  Immediate  Targeting 
Questionnaire  (ITQ)  are  those  in  the  top  (speckled)  portion  of  the 
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slide.  Factors  used  to  generate  the  Target  Identification  Questionnaire 
(TIQ)  are  those  in  the  lower  (dotted)  portion  of  the  slide.  For  both 
questionnaires,  the  scenario  used  as  a  background  was  the  Korean 
scenario  described  in  Volume  III  of  the  Air  Force  Planning  Guide. 

Before  being  given  the  questionnaires,  respondents  were  given  a  copy  of 
the  scenario  and  were  briefed  on  the  factors  comprising  the  immediate 
targeting  structure. 

Respondents  for  both  questionnaires  were  Air  Force  officers 
assigned  to  the  Pacific  Air  Forces  (PACAF)  and  Headquarters,  Pacific 
Command.  They  participated  on  an  as-available  basis  through  informal 
cooperation  with  the  PACAF  Assistant  DCS  for  Intelligence  and  the 
Special  Assistant  for  Operations  Analysis,  DCS/Operations  and 
Intelligence. 

The  next  two  slides  describe  the  factors  and  factor  levels 
separately  for  the  two  questionnaires. 
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IMMEDIATE  TARGETING  QUESTIONNAIRE 
FACTOR  LEVELS 


|  Immediate  targeting 


%  of  affective  force  application  opportunities  exploited 


Status 

Status 

Currency 

access 

access 

90% 

90% 

1  5  mm 

60% 

60% 

1  hr 

30% 

30% 

3  hrs 

10% 

10% 

12  hrs 
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The  factors  and  factor  levels  used  in  the  ITQ  are  shown  here. 
Questionnaire  items  were  comprised  of  different  combinations  of  the 
factor  levels.  Thus,  each  item  described  a  different  set  of  system 
capabilities.  For  each  item,  respondents  judged  the  percent  of  the 
effective  force  application  opportunities  that  could  be  exploited  by  the 
immediate-targeting  function,  given  a  command  and  control  system  that 
had  the  described  capabilities.  Effective  force  application 
opportunities  were  defined  as  resulting  from  the  true  battle  situation 
and  the  true  availability  of  forces,  whether  or  not  they  could  be 
perceived  by  the  immediate-targeting  decisionmakers.  An  effective  force 
application  would  be  the  matching  of  a  proper  tactical  air  weapon  system 
with  an  important  enemy  target  at  an  appropriate  time.  The  practical 
advantages  of  this  outcome  measure  are  that  it  can  be  used  to  indicate 
relative  values,  in  an  operational  context,  of  different  configurations 
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of  systems  that  support  the  immediate-targeting  function,  as  well  as 
provide  a  valid  subjective  input  value  to  force  effectiveness  utility 
models  in  the  MAA  theater  engagement  mission  area. 

The  input  factor.  Enemy  Second  Echelon  Force  Information,  reflects 
the  percentage  of  the  second  echelon  force  elements  important  to  the 
course  of  battle  that  are  identified  in  time  for  use  in  performing  the 
immediate-targeting  function.  The  factor  levels  range  from  10%  to  90%-- 
a  range  considered  sufficient  to  span  all  feasible  system 
configurations . 

The  input  factor,  Facility  Operability,  reflects  the  ability  of  the 
tactical  air  control  facilities  to  support  the  immediate-targeting 
function.  It  includes  condiderations  of  survivability,  capacity,  and 
endurance.  It  is  described  in  terms  of  the  percentage  of  all  the 
immediate-targeting  activities  that  would  be  necessary  for  the  facility 
to  perform  if  all  effective  force  application  opportunities  were  to  be 
exploited.  Again,  the  factor  levels  range  from  10%  to  90%. 

Dissemination  reflects  the  ability  to  communicate  tasking  orders  to 
the  tactical  air  forces.  It  is  described  in  terms  of  the  percentage  of 
forces  to  which  tasking  orders  can  correctly  be  communicated  in  time  for 
those  forces  to  accomplish  the  tasking.  Forces  of  interest  to  the 
immediate -targeting  function  are  those  on  the  ground  and  in  the  air  that 
can  be  scrambled  or  diverted,  respectively,  to  second  echelon 
interdiction  missions.  Factor  levels  range  from  10%  to  90%  of  the 
available  forces. 

Execution  Status  Information  reflects  the  status  of  our  friendly 
forces.  It  is  proposed  that  three  factors  affect  Execution  Status 


Information.  The  input  factor,  Alert  Forces,  reflects  the  ability  to 
access  information  about  the  status  of  the  designated  alert  forces. 
Similarly,  the  input  factor.  Airborne  Forces,  reflects  the  ability  to 
access  information  about  the  status  of  tactical  air  forces  that  are 
airborne.  Both  factors  are  described  in  terms  of  the  percentage  of  the 
forces  about  which  status  information  can  be  accessed  in  time  to  perform 
the  immediate-targeting  function.  The  factor  levels  range  from  10%  to 
90%. 

The  input  factor,  Weather,  reflects  the  currency  of  reliable 
weather  data  about  the  enemy  second  echelon  area  and  the  tactical  air 
bases.  The  factor  levels  range  from  12  hours  to  15  minutes. 

The  factors  and  factor  levels  used  in  the  ITQ  are  described  in  more 
detail  in  Appendix  A. 
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Slide  9 


This  slide  depicts  the  factors  and  factor  levels  used  in  the  TIQ. 


The  outcome  measure  for  this  questionnaire  (relating  to  enemy  second 
echelon  force  information)  is  defined  along  the  same  continuum  as  when 
this  component  was  used  as  an  input  factor  in  the  ITQ  (see  Slide  8). 
That  is,  respondents  judged  the  percent  of  the  important  second  echelon 
force  elements  that  could  be  identified  in  a  timely  manner. [2] 

This  outcome  measure  is  like  the  outcome  measure  for  the  ITQ  in 


that  it  can  be  used  to  indicate  relative  values,  in  an  operational 
context,  of  different  configurations  of  systems  that  support  the 
tactical  fusion  portion  of  the  immediate-targeting  function.  It  also 


[2]  In  the  STF  approach,  this  feature  of  using  a  factor  as  both  an 
independent  and  dependent  variable  serves  to  functionally  link  major 
sections  of  hierarchical  structures  (see  Veit  and  Callero,  1981,  for 
more  detail).  The  Enemy  Second  Echelon  Force  Information  transfer  func 
tion  (T ( 3 ) )  serves  to  link  the  two  major  sections--immediate  targeting 
and  target  identif ication--of  our  immediate-targeting  structure.  This 
will  be  illustrated  in  the  results  section. 
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can  be  used  as  input  to  force  effectiveness  utility  models  in  the  MAA 
theater  engagement  mission  area. 

The  input  factors  at  the  top  of  this  portion  of  the  immediate- 
targeting  structure  are  data  about  enemy  vehicles  (Vehicles);  data  about 
enemy  emitters  (Emitters);  and  the  process  by  which  these  data  are 
analyzed  (Processing). 

The  factor  levels  associated  with  the  three  factors  hypothesized  to 
affect  enemy  vehicle  information,  T(4),  are  as  follows.  The 
location/classification  input  factor  that  reflects  the  ability  of  our 
sensor  systems  to  locate  and  classify  enemy  vehicles  has  four  factor 
levels : 

1.  Locate  and  classify  vehicles  in  all  weather  conditions. 

2.  Locate  (but  not  classify)  vehicles  in  all  weather  conditions. 

3.  Locate  and  classify  vehicles  in  clear  weather  only. 

4.  Locate  (but  not  classify)  vehicles  in  clear  weather  only. 

The  coverage  input  factor  reflects  the  percentage  of  enemy  vehicles  in 
the  second  echelon  area  that  have  been  observed.  The  factor  levels 
range  from  10%  to  90%.  The  currency  input  factor  reflects  the  time 
interval  between  observing  vehicles  in  the  second  echelon  area  and  the 
availability  of  the  data  for  processing  in  the  command  and  control 
system.  The  factor  levels  range  from  1  hour  to  5  minutes.  Times  beyond 
1  hour  were  considered  not  relevant  to  the  immediate-targeting  function. 

The  factor  levels  associated  with  the  three  factors  hypothesized  to 
impact  on  the  capability  to  pick  up  enemy  emitter  information  (T(5))  are 
as  follows.  The  input  factor  levels  for  Location  refer  to  accuracy  with 
which  enemy  emitters  can  be  located;  they  vary  from  within  10m  to  within 
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1000m.  The  input  factors  Coverage  and  Currency  have  the  same 
interpretation  and  factor  levels  as  for  vehicles. 

Processing  reflects  the  means  by  which  enemy  vehicle  and  emitter 
data  are  interpreted.  The  factor  levels  vary  by  extent  of  computerized 
support.  At  one  level  there  is  no  automation;  humans  interpret  the  data 
by  sorting  through  hard-copy  textual  materials  (e.g.,  formatted 
reports).  The  other  three  levels  include  automation  and  computer  output 
devices.  At  one  level  the  human  uses  the  computer  to  organize,  select, 
search  and  display  textual  data  to  facilitate  interpretation.  At 
another  level,  the  human  also  uses  automated  graphic  displays  to  present 
pictures  made  up  of  symbols  and  line  drawings,  for  example,  a  map  with 
military  units,  towns,  roads,  and  terrain  features.  At  the  fully 
computerized  level,  the  computer  alone  organizes  and  interprets  the 
data. 

The  factors  and  factor  levels  used  in  the  TIQ  are  described  in  more 
detail  in  Appendix  B. 
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QUESTIONS  ADDRESSED  IN  STF  APPROACH 


•  Are  all  factors  included  relevant? 


•  Is  the  structure  appropriate? 


•  What  are  the  appropriate  transfer  functions? 


Slide  10 


Once  an  initial  structure  of  the  problem  domain  has  been 
hypothesized,  the  STF  approach  is  designed  to  anther  the  following 
questions : 

(1)  Do  each  of  the  hypothesized  factors  empirically  affect  the  Air 
Force  professionals'  judgments;  that  is,  have  irrelevant  factors  been 
included  in  the  structure? 

(2)  Is  the  structure  appropriately  depicted  by  five  causal  sets 
and  hence  five  transfer  functions:  two  for  the  immediate-targeting 
portion  of  the  structure  and  three  for  the  target  identification 
portion? 

(3)  What  are  the  appropriate  transfer  functions  that  best  explain 
the  j udgment  data ;  that  is,  what  are  the  functional  rules  that  Air  Force 
professionals  use  in  making  their  immediate  targeting  and  important 
target  identification  judgments? 

The  questionnaires  fielded  for  study  were  designed  to  answer  these 
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questions.  In  the  STF  approach,  these  three  questions  are  addressed 
simultaneously  with  tests  of  the  hypothesized  STFs. 

Once  answers  to  the  questions  are  found,  it  is  possible  to  use  the 
STFs  to  assess  the  capabilities  of  different  command  and  control  systems 
described  by  the  structure. 
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This  slide  describes  a  number  of  different  algebraic  models  that 
might  serve  as  STFs.  Each  model  makes  different  predictions  (described 
in  more  detail  in  the  results  section)  with  respect  to  a  set  of  judgment 
data.  It  was  necessary  to  consider  this  wide  variety  of  models  because 
judgment  research  had  never  been  performed  on  the  factors  and  factor 
levels  entertained  in  our  immediate-targeting  structure  (Slides  8  and 
9). 

Research  designs  selected  to  generate  the  questionnaires  allowed 
tests  among  the  additive  and  interactive  models  outlined  on  this  slide. 

A  major  questionnaire  design  feature  used  to  test  among  the  predictions 
of  these  models  is  the  factorial  combination  of  the  hypothesized  factor 
levels . 

For  both  questionnaires,  items  were  generated  from  factorial 
combinations  of  the  input  factor  levels.  Each  item  represented  a 
different  set  of  command  and  control  capabilities  to  which  the  Air  Force 
professional  was  asked  to  respond  in  a  particular  manner  (described 
next).  Details  of  the  designs  used  to  generate  the  ITQ  and  TIQ  are 
presented  in  Appendixes  A  and  B,  respectively. 
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JUDGMENTS  OF  IMMEDIATE  TARGETING 
EFFECTIVENESS 


1 92  questions;  1 3  respondents;  40-60  minutes 

Response  scale:  %  of  the  effective  force  application  opportunities  that 

could  be  exploited  by  the  Immediate  Targeting  Function 

Sample  Items: 

Weather  data  are  1 5  min  old 

Facilities  can  support  60%  of  the  necessary  l/T  activities 
Timely  access  to  the  status  of  90%  of  the  designated  Alert  forces 
Timely  access  to  the  status  of  30%  of  the  Airborne  forces 

Weather  data  are  12  hrs  old 

Timely  access  to  the  status  of  90%  of  the  Airborne  forces 
Tasking  can  be  correctly  communicated  to  60%  of  the  forces  in  time 
Timely  access  to  the  status  of  1 0%  of  the  designated  Alert  forces 

Slide  12 


For  the  ITQ,  13  respondents  completed  192  items.  Respondents  were 
instructed  to  treat  each  item  as  a  different  situation;  that  is,  each 
item  represented  a  different  description  of  the  command  and  control 
capabilities.  For  each  item,  respondents  judged  the  percent  of 
effective  force  application  opportunites  that  could  be  exploited. 
Respondents  took  from  40  to  60  minutes  to  complete  the  questionnaire. 
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JUDGMENTS  OF  IMPORTANT  TARGET 
IDENTIFICATION 


203  questions;  1 3  respondents;  40-60  minutes 

Response  scale:  %  of  important  second  echelon  force 
elements  that  could  be  identified 
in  a  timely  manner 

Sample  items: 

Emitters  are  located  within  a  10m  accuracy 
Emitter  info  is  available  for  processing  5  min  after  observation 
Human  sorts  hard  copy,  textual  info;  Human  interpretation 
10%  of  the  second  echelon  Emitters  are  observed 

Emitter  info  is  available  for  processing  5  min  after  observation 
Human  uses  computer  to  sort  textual  info;  Human  interpretation 
90%  of  the  second  echelon  Emitters  are  observed 
Emitters  are  located  within  a  100m  accuracy 

Slide  13 


Instructions  and  procedures  were  similar  for  respondents  answering 
the  TIQ.  For  each  item,  respondents  judged  the  percent  of  important 
second  echelon  force  elements  that  could  be  identified  in  the  given 
situation.  Thirteen  respondents  took  between  40  to  60  minutes  to 
complete  the  203  items. 
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RESULTS 


The  goal  in  data  analyses  was  to  find  the  set  of  transfer  functions 
(algebraic  models)  that  best  explained  the  judgment  data  and  thus  the 
immediate-targeting  structure. 

Below,  we  discuss  the  results  for  the  two  questionnaires  (slides 
14-21).  For  both  sets  of  analyses,  judgments  were  averaged  over  all 
13  respondents.  Model  predictions  were  tested  against  mean  judgments. 
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Data  from  ITQ 


10%  30%  60%  90% 

Dissemination 
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ANALYSES  AND  RESULTS  FOR  THE  IMMEDIATE -TARGETING  QUESTIONNAIRE  DATA 

This  slide  presents  representative  graphic  and  statistical  analyses 
of  the  ITQ  data. 

In  each  panel,  mean  judgments  (percents)  of  immediate-targeting 
opportunities  that  could  be  exploited  are  plotted  on  the  y-axis  as  a 
function  of  one  of  the  immediate-targeting  factors  on  the  x-axis.  A 
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separate  curve  is  plotted  for  each  level  of  one  of  the  other  immediate 
targeting  factors.  The  slopes  of  the  curves  in  each  panel  indicate  the 
main  effect  of  the  factor  plotted  on  the  x-axis;  vertical  separations 
between  the  curves  depict  the  main  effect  of  the  other  factor.  Every 
factor  included  in  the  ITQ  had  a  significant  effect  on  judged  outcomes. 
This  finding  argues  for  their  inclusion  in  the  immediate  targeting 
structure. 

Both  graphic  and  statistical  (analysis  of  variance)  analyses  of  the 
ITQ  data  revealed  interactions  among  a  number  of  the  factors.  That  is, 
judgments  tended  to  exhibit  synergistic  effects.  These  particular 
interaction  effects  are  divergent  in  nature  and  can  be  seen  by  examining 
the  data  points  in  the  separate  panels  above.  As  capability  associated 
with  one  factor  (e.g.,  Facility  Operability  in  the  upper  right-hand 
panel)  improved,  the  capability  level  of  the  other  factor  (e.g. ,  Enemy 
Information)  made  more  of  a  difference  (i.e.,  had  a  greater  effect). 
(This  can  be  seen  by  comparing  the  difference  in  the  vertical  distance 
between  the  top  and  bottom  curves  at  10%  and  at  90%  Facility  Operability 
in  the  upper  right-hand  panel.  The  same  effect  can  be  seen  in  the  other 
panels.)  These  interactions  ruled  out  additive  and  averaging  models  as 
an  appropriate  transfer  function  at  the  top  of  the  structure.  Further, 
the  observed  divergence  was  not  of  the  form  that  would  be  expected  if 
values  placed  on  factor  levels  were  being  subjectively  multiplied. [ 1] 
Thus,  all  models  that  specified  multiplicative  interactions  among 
factors  were  also  rejected  as  appropriate  transfer  functions. 

[1]  Multiplicative  models  predict  bilinear  interactions  among  fac¬ 
tors  hypothesized  to  combine  multiplicatively  (an  example  is  shown  in 
Veit  and  Callero,  1981). 
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A  range  model  (shown  on  slide  11)  captured  the  particular 
synergistic  effects  observed  in  the  judgments.  A  range  model  predicts 
that  the  extremity  of  the  information  contained  in  the  questionnaire 
items  has  a  unique  effect  on  the  judged  outcomes.  For  these  divergent 
interactions,  the  range  model  had  a  negative  range  coefficient  (omega  in 
Slide  11),  indicating  the  increased  effect  of  one  factor  with  the 
improvement  in  capability  of  another  factor. 

Further,  the  range  model  that  best  explained  the  Immediate 
Targeting  judgments  was  a  six-factor  model  that  included  the  three 
execution  status  information  factors--Alert  Forces,  Airborne  Forces,  and 
Weather--as  separate  factors  in  the  model.  That  is,  the  best 
explanation  of  the  data  was  one  that  eliminated  execution  status 
information  as  a  separate  causal  set  in  the  hierarchical  structure. 

These  results  argue  for  collapsing  the  top  two  tiers  of  the  immediate- 
targeting  structure  and  depicting  this  portion  of  the  structure  as  a 
single  causal  set  of  six  factors --Enemy  Second  Echelon  Force 
Information,  Facility  Operability,  Alert  Forces,  Airborne  Forces, 

Weather,  and  Dissemination--impacting  on  Immediate  Targeting.  The 
average  data/model  discrepancy  over  all  192  data  points  was  less  than  4 
percent  for  this  single  six-factor  range  model. 

The  single  six-factor  range  model  was  tested  against  alternative 
possible  explanations  of  the  data.  (Competing  sets  of  models  were 
tested  using  a  computer  program  (written  at  Rand)  that  used  the  STEPIT 
subroutine[2]  to  find  a  simultaneous  least-squares  solution  to  all  192 
data  points.)  Because  a  divergent  interaction  of  the  form  shown  in  the 

[2]  J.  P.  Chandler,  "STEPIT- -Finds  Local  Minima  of  a  Smooth  Function 
of  Several  Parameters,"  Behavioral  Science ,  Vol.  14,  1969,  pp.  81-82. 
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above  panels  was  found  between  the  factors  Airborne  Forces  and  Weather, 
the  immediate-targeting  portion  of  the  structure  was  also  treated  as  a 
two-transfer  function  (two  causal  set)  structure  with  a  range  model  as 
the  transfer  function  for  execution  status  information  and  a  range  model 
for  immediate  targeting. [3]  The  two  range  models,  however,  did  more 
poorly  than  the  single  six-factor  range  model  in  predicting  the  data. [4] 

Interactions  among  the  other  pairs  of  execution  status  information 
factors  and  the  three-way  interaction  were  not  significant.  Thus,  the 
relative-weight  averaging  model  with  an  initial  impression  shown  in 
Slide  11  was  also  entertained  as  a  transfer  function.  Again,  the  fit 
for  a  two-transfer  function  structure  with  a  range  model  for  immediate 
targeting  and  a  relative-weight  averaging  model  with  an  initial 
impression  for  execution  status  information  was  poorer  than  the  single 
six-factor  range  model. [5] 

For  both  two-transfer  function  representations,  the  number  of 
parameters  estimated  was  greater  and  the  models'  fits  were  poorer. 


[3]  This  two-transfei  function  structure  was  tested  by  fitting  a 
range  model  to  the  data  obtained  from  the  designs  (see  Appendix  A)  that 
included  just  the  execution  status  information  factors,  and  embedding 
the  outputs  obtained  from  that  function  in  a  range  model  fitted  to  data 
from  designs  that  included  all  the  factors. 

[4]  The  sum  of  squared  data/model  discrepancies  was  44.6  for  the 
two  range  models  (with  35  estimated  parameters)  and  28.8  (with  33  es¬ 
timated  parameters)  for  the  single  six-factor  range  model. 

[5]  The  sum  of  squared  data/model  deviations  for  the  two-transfer 
function  structure  (a  range  model  for  immediate  targeting  and  a 
relative -weight  averaging  model  with  an  initial  impression  for  execution 
status  information)  was  48.5  with  34  estimated  parameters). 


29 


Theoretical  predictions 


Slide  15 


The  judgment  theory  provides  the  basis  for  making  predictions . [6] 
This  slide  illustrates  how  subjective  tradeoffs  in  factor  levels  can  be 
assessed  once  the  STFs  have  been  determined.  Points  on  these  curves  are 
the  theoretical  predictions  (subjective  values  derived  from  the  six- 
factor  range  model)  for  the  immediate-targeting  data  shown  in  the  upper 
left-hand  panel  of  Slide  14.  Theoretical  values  (subjective  response 
values  derived  from  the  range  model)  are  plotted  on  the  y-axis  as  a 
function  of  the  dissemination  factor.  (The  spacing  on  the  x-axis 


[6]  The  best  predictor  of  data  points  will  be  the  causal  theory 
that  explains  their  behavior.  For  example,  in  astronomy,  movements  of 
the  planets  are  predicted  better  from  the  theory  that  explains  their  or¬ 
bital  movements  than  from  empirical  tables  of  previous  data  points. 
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reflects  the  subjective  spacing  between  the  dissemination  factor  levels; 
their  corresponding  physical  values  are  noted  on  the  graph.)  A  separate 
curve  is  plotted  for  each  level  of  Facility  Operability.  The 
theoretical  predictions  can  be  used  to  assess  the  subjective  tradeoffs 
between  these  two  factors  by  drawing  horizontal  lines  through  the 
curves.  For  example,  the  subjective  value  of  the  percent  of  immediate- 
targeting  opportunities  that  could  be  exploited  seems  to  be  about  the 
same  for  a  Dissemination  capability  of  90%  and  a  Facility  Operability 
capability  of  30%,  and  a  Dissemination  capability  of  30%  and  a  Facility 


Operability  capability  of  60%.  Another  way  of  looking  at  this  is  the 
following.  Suppose  your  present  capability  lies  at  the  30%  Facility- 
Operability  and  30%  Dissemination  point.  You  could  do  about  equally 
well  in  increasing  your  overall  capability  either  by  increasing  your 
Facility  Operability  to  60%  or  by  increasing  your  Dissemination 
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In  the  STF  approach,  once  an  appropriate  STF  has  been  found, 
subjective  scale  values  (utilities)  associated  with  the  factor  levels 
can  be  derived  from  the  STF.  When  the  factor  levels  are  physical  values 
(such  as  the  percents  used  in  this  study),  the  function  relating  the 
subjective  values  to  the  physical  values  can  be  determined. 

This  slide  demonstrates  the  functional  relationship  between 
subjective  scale  values  associated  with  the  factor  levels  and  their 
physical  descriptors.  In  each  panel,  subjective  scale  values  derived 
from  the  range  model  are  plotted  on  the  y-axis.  The  physical  values 
associated  with  the  factor  levels  are  plotted  on  the  x-axis.  Each  panel 
is  for  a  different  factor  used  in  the  ITQ  that  had  numerical  factor- 
level  descriptions.  These  functions  (referred  to  as  psychophysical 
functions  or  utility  curves)  are  all  nonlinear. 

Estimates  of  the  subjective  scale  values  associated  with  each  level 
of  the  six  immediate-targeting  factors,  subjective  weights  associated 
with  each  of  the  six  factors,  and  the  range  coefficient  are  presented  in 
Appendix  C. 
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SUMMARY  OF  ITQ  RESULTS 


•  All  hypothesized  factors  are  relevant 

•  Altered  immediate  targeting  structure 


•  Transfer  function:  6 -factor  range  model 

Slide  17 

In  summary,  the  major  conclusions  for  the  data  obtained  from  the 
ITQ  are: 

(1)  All  manipulated  factors  should  be  included  in  the  structure. 

(2)  A  better  fit  to  the  data  was  achieved  by  describing  Execution 
Status  Information  as  three  separate  factors  rather  than  a  single 
composite  factor,  which  argues  for  excluding  this  causal  set  from  the 
immediate-targeting  structure;  the  altered  structure  is  shown  on  this 
slide. 

(3)  A  six-factor  range  model  with  a  negative  range  coefficient 


provided  the  best  fit  to  the  data,  which  argues  for  this  model  as  the 
transfer  function  for  immediate  targeting. 


Mean  judgment  (percent 
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Data  fromTIQ 


Co* -10%  Co* -80%  Co* -90%  Co* -90% 

Curr - 1  hr  Curr-lhr  Curr-lhr  Curr -5  min 

Vehicle  combinations 


Emitter  currency  and  coyer  age 
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ANALYSES  AND  RESULTS  FOR  IMPORTANT  TARGET  IDENTIFICATION  QUESTIONNAIRE  DATA 
The  four  panels  shown  in  this  slide  depict  data  obtained  from  the 
TIQ.  In  each  panel,  mean  judgment  (percent)  of  important  second  echelon 
force  elements  that  could  be  identified  is  plotted  as  a  function  of  one 
factor,  with  a  separate  curve  for  each  level  of  another  factor.  Again, 
the  slopes  of  the  curves  in  each  panel  of  this  slide  depict  the  main 
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effect  of  the  factor  on  the  x-axis;  vertical  separations  between  the 
curves  represent  the  main  effect  of  the  other  factor. 

As  with  factors  included  in  the  ITQ,  each  factor  included  in  the 
TIQ  had  a  significant  main  effect  on  the  judgments,  which  argues  for 
inclusion  of  all  initially  hypothesized  factors  in  the  immediate- 
targeting  structure. 

The  trends  found  in  the  TIQ  data  were  markedly  different  for  each 
of  the  three  causal  sets  hypothesized  to  comprise  the  target 
identification  portion  of  the  immediate-targeting  structure.  These 
trends  are  graphically  depicted  in  the  lower  left-hand  corner  for  enemy 
second  echelon  force  information  data;  the  upper  left-hand  corner  for 
vehicle  data;  and  the  two  right-hand  panels  for  emitter  data.  These 
different  judgment  trends  were  best  accounted  for  by  a  separate  transfer 
function  for  each  of  the  three  hypothesized  causal  sets,  thus  supporting 
the  idea  that  all  three  sets  should  be  included  in  the  final  immediate- 
targeting  structure. 

We  will  describe  the  trends  in  the  data  separately  for  each  causal 
set  as  well  as  the  transfer  function  that  was  supported  by  the  data. 

Then  we  will  discuss  the  alternative  transfer  functions  and  structures 
that  were  considered. 

Enemy  Second  Echelon  Force  Information  Data 

Questionnaire  items  that  corresponded  to  this  causal  set  in  the 
immediate-targeting  structure  contained  information  about  the  three 
factors — Vehicles,  Processing,  and  Emitters--hypothesized  to  directly 
impact  on  enemy  second  echelon  force  information  capabilities.  That  is, 
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the  items  contained  information  on  enemy  vehicle  data,  processing 
capabilities,  and  enemy  emitter  data. 

In  the  lower  left-hand  panel  of  this  slide,  vehicle  capability  is 
plotted  on  the  x-axis  (each  point  on  the  x-axis  represents  a  different 
combination  of  vehicle  Location,  Coverage,  and  Currency  levels).  Four 
separate  curves  are  plotted  for  four  different  combinations  of  emitter 
capability  (different  combinations  of  Location,  Coverage,  and  Currency 
levels).  It  can  be  seen  from  this  graph  that  both  convergent  and 
divergent  interaction  trends  are  occurring  in  the  data.  If  only  the  top 
three  curves  are  considered,  the  curves  diverge  from  the  second  vehicle 
combination  (second  value  on  the  x-axis)  to  the  last  vehicle 
combination.  However,  when  the  bottom  curve  is  considered,  a  marked 
convergence  occurs  between  that  curve  and  the  other  three  curves  from 
the  lowest  to  the  highest  value  on  the  x-axis.  That  is,  when  the 
capability  of  picking  up  information  on  enemy  vehicles  is  very  poor 
(lowest  point  on  the  x-axis),  it  makes  a  big  difference  as  to  how  well 
enemy  emitter  data  can  be  picked  up.  As  the  capability  of  picking  up 
vehicle  data  improves  (moving  to  the  right  on  the  x-axis),  this 
difference  is  less.  Overall,  the  interaction  between  these  two  factors 
was  not  statistically  significant  but  it  was  systematic. 

The  same  divergent/convergent  interaction  patterns  were  found  for 
the  other  enemy  second  echelon  force  information  data.  The  interaction 
between  Processing  and  Emitters  was  slightly  divergent  (of  borderline 
significance),  while  the  three-way  interaction  between  Vehicles, 
Processing  and  Emitters  was  convergent.  Even  though  the  interactions 
were  either  nonsignificant  or  of  borderline  significance,  a  range  model 


37 


with  a  positive  range  coefficient  best  explained  these  data  (described 
in  more  detail  below). 

Vehicle  Data 

The  graph  shown  in  the  upper  left-hand  corner  of  the  slide  depicts 
a  significant  divergent  interaction  between  the  percent  of  enemy 
vehicles  observed  (Coverage)  and  the  currency  of  this  vehicle 
information  (how  quickly  it  gets  into  the  command  and  control  system). 
Note  that  it  makes  more  of  a  difference  to  respondents  how  much 
information  comes  into  the  command  and  control  system  when  the  data  are 
current  than  when  they  are  not  current  (compare  the  magnitude  of  the 
vertical  separations  between  the  curves  at  the  lowest  and  highest  value 
on  the  x-axis).  This  divergent  interaction  was  typical  of  the 
relationship  observed  among  all  of  the  vehicle  factors.  These  prevalent 
divergent  interactions  ruled  out  additive  models  to  explain  the  data. 
Multiplicative  models  were  also  ruled  out  because  the  interactions  did 
not  have  the  bilinear  form  predicted  by  multiplicative  models.  A  range 
model  with  a  negative  range  coefficient  best  accounted  for  these  data 
(discussed  below). 

Emitter  Data 

In  the  upper  right-hand  panel  of  the  slide,  the  Currency  of  enemy 
emitter  information  is  plotted  on  the  x-axis;  a  separate  curve  is 
plotted  for  each  level  of  emitter  Location  capability  (the  accuracy  with 
which  enemy  emitters  could  be  located.)  Note  that  the  top  and  bottom 
curves  are  quite  parallel  (the  class  of  additive  models  predicts 
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parallelism  among  factors).  However,  the  middle  curve  representing  an 
emitter  location  accuracy  of  100m  tends  to  interact  with  both  the  top 
and  bottom  curves  but  in  different  ways.  This  curve  converges  with  the 
top  and  diverges  with  the  bottom  curve.  This  interaction  was  not 
significant.  The  other  two-way  interaction  (between  emitter  location 
and  coverage)  and  the  three-way  interaction  were  also  nonsignificant. 

The  single  significant  interaction  was  a  divergent  interaction  between 
the  emitter  coverage  and  currency  factors.  However,  the  model  that  best 
explained  all  these  data  was  the  relative-weight  averaging  model  with  an 
initial  impression  (shown  in  Slide  11). 

The  data  shown  in  the  lower  right  hand  panel  of  this  slide  are 
presented  to  demonstrate  why  a  simple  additive  model  (the  first  model 
shown  on  Slide  11)  was  ruled  out  for  the  emitter  data.  The  additive 
model  predicts  that  the  effect  of  a  factor  (e.g. ,  Emitter  Location 
Accuracy)  should  be  independent  of  the  number  of  other  pieces  of 
information  describing  the  situation  to  be  judged.  Compare  the 
magnitudes  of  the  separations  between  the  two  curves  in  the  two 
subpanels.  Note  that  the  effect  of  emitter  location  accuracy  is  less 
when  three  pieces  of  information  (information  about  emitter  location 
accuracy,  percent  of  emitters  observed  (Emitter  Coverage),  and  currency 
of  emitter  information)  describe  the  situation  to  be  judged  (right-hand 
subpanel)  than  when  only  two  pieces  of  information  describe  the 
situation  to  be  judged.  That  is,  the  additional  information  decreased 
the  effect  of  a  particular  piece  of  information- -Emitter  Location 
Accuracy.  This  diminished  effect  infirms  an  additive  model  but  is 
predicted  by  an  averaging  model. 
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Tests  of  TIQ  Transfer  Functions  and  Structures 

As  noted  in  the  last  three  sections,  a  three-transfer  function 
structure  was  supported  by  the  data;  the  three  functions  considered 
appropriate  were:  (1)  a  range  model  with  a  positive  range  coefficient 
(to  pick  up  the  convergent  trend  shown  in  the  lower  left-hand  panel  of 
this  slide),  which  was  the  best  transfer  function  among  those 
entertained  for  enemy  second  echelon  force  information  data;  (2)  a  range 
model  with  a  negative  range  coefficient  (to  pick  up  the  divergent 
interaction  trends  illustrated  in  the  upper  left-hand  panel),  which  was 
the  best  transfer  function  for  the  vehicle  data;  and  (3)  a  relative- 
weight  averaging  model  (to  account  for  the  primarily  additive  emitter 
data),  which  was  the  best  transfer  function  for  the  emitter  data. [7]  The 
average  data/model  discrepancy  over  all  203  data  points  for  this 
combination  was  about  4  percent. [8] 

We  tested  the  notion  that  this  portion  of  the  immediate-targeting 
structure  should  be  represented  by  only  one  causal  set,  by  assessing  how 
well  a  single  seven- factor  model  accounted  for  the  data.  Three  seven- 
factor  models  for  a  single-transfer  function  structure  were  entertained. 
These  were:  (1)  a  range  model,  (2)  a  relative-weight  averaging  model, 
and  (3)  a  relative-weight  averaging  model  with  an  initial  impression 
(Eqs.  5,  2,  and  3,  respectively,  on  Slide  11).  All  of  these 
possibilities  provided  a  poorer  fit  to  the  data  than  the  three-transfer 
function  structure  described  above. [9] 

[7]  This  combination  of  models  was  tested  by  embedding  the  output 
of  the  vehicle  range  model  and  the  output  of  the  emitter  relative-weight 
averaging  model  in  the  enemy  second  echelon  force  information  range 
model . 

[8]  The  sum  of  squared  data/model  errors  for  this  set  of  models  was 
34.8  with  40  paramters  estimated. 

[9]  The  sum  of  squared  data/model  deviations  for  the  range  model, 
relative-weight  averaging  model,  and  relative-weight  averaging  model 
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We  also  assessed  two  different  three-transfer  function  (three 
causal  set)  hypotheses  based  on  the  patterns  observed  in  the  data. [10] 
First,  we  tested  the  hypothesis  that  a  range  model  was  appropriate  for 
the  emitter  data  (to  pick  up  the  emitter  Location  and  Currency 
interaction  trends  shown  in  the  upper  right-hand  panel);  a  range  model 
was  appropriate  for  the  vehicle  data  (to  pick  up  the  divergent 
interactions);  and  a  relative-weight  averaging  model  with  an  initial 
impression  was  appropriate  for  the  enemy  second  echelon  force 
information  data  (interactions  among  factors  here  were  borderline  or 
nonsignificant).  This  three -function  hypothesis  did  poorer  than  the 
accepted  three-function  hypothesis .[ 1 1 ]  Our  second  three- function 
hypothesis  was  like  the  first  in  that  a  range  model  was  tested  for  the 
emitter  and  vehicle  data,  but  different  in  that  a  relative-weight 
averaging  model  without  an  initial  impression  (see  Eq.  2,  Slide  11)  was 
tested  for  the  enemy  second  echelon  force  information  data.  When  the 
initial  impression  was  excluded  from  the  averaging  model  for  enemy 
second  echelon  force  information  data,  the  sum  of  squared  data/model 
deviations  got  substantially  worse. [12] 

A  table  of  subjective  scale  values,  weights,  and  range  coefficients 
derived  from  the  three-transfer  function  combination  described  in  the 

first  paragraph  of  this  section  is  presented  in  Appendix  D. 

with  an  initial  impression  were  51.3,  127.5,  and  41.2,  respectively, 
with  37,  34,  and  36  estimated  parameters,  respectively. 

[10]  These  three-transfer  function  hypotheses  were  tested  by  embed¬ 
ding  the  outputs  from  the  emitter  and  vehicle  functions  in  the  enemy 
second  echelon  force  information  model. 

[11]  This  competing  combination  had  a  sum  of  squared  errors  of  59.2 
with  40  estimated  parameters. 

[12]  The  sum  of  squared  deviations  increased  to  126.3.  It  is  in¬ 
teresting  to  note  that  the  averaging  model  without  an  initial  impression 
is  the  model  that  is  often  accepted  as  appropriate  for  all  functions  in 
complex  systems  analyses  without  empirical  tests  (of  factors,  structure, 
or  the  function).  When  this  model  was  tested  for  all  three  causal  sets 
in  this  portion  of  the  structure,  the  sum  of  squared  data/model  devia¬ 
tions  was  126.3  with  36  estimated  parameters. 
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Theoretical  predictions 


Slide  19 


This  slide  illustrates  how  subjective  tradeoffs  in  factor  levels 
can  be  assessed  once  the  STFs  have  been  determined.  In  this  graph, 
subjective  values  associated  with  enemy  emitter  information  (derived 
from  the  accepted  three-transfer  function  combination)  are  plotted  on 
the  y-axis.  The  four  levels  of  emitter  currency,  spaced  according  to 
their  subjective  values,  are  on  the  x-axis.  There  is  a  separate  curve 
for  each  level  of  the  emitter  location  factor. 

This  graph  tells  us,  for  example,  that  an  emitter  location  accuracy 
capability  of  1000m  combined  with  an  emitter  currency  capability  of  5 
minutes  is  subjectively  equivalent  to  a  location  accuracy  capability  of 


100m  combined  with  a  currency  capability  of  15  minutes.  Also,  a 
location  accuracy  capability  of  100m  combined  with  a  currency  capability 
of  5  minutes  is  about  equal  to  a  location/currency  combination  of  10m 
and  about  20  minutes  (interpolated  from  the  graph).  Other  subjective 
tradeoffs  can  be  assessed  by  drawing  horizontal  lines  through  the 
curves . 
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Utility  curves:  TIQ 


1  hr  30  min  15  min  5  min 

Emitter  currency 
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This  slide  depicts  the  form  of  the  psychophysical  functions 
(utility  curves)  obtained  for  those  factors  comprising  the  TIQ  that  had 
factor  levels  measured  on  a  physical  scale.  Once  these  functions  are 
known,  subjective  scale  values  for  factor  levels  not  used  in  the  study 
can  be  determined  from  the  functions. 
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SUMMARY  OF  TIQ  RESULTS 


•  All  hypothesized  factors  are  relevant 

•  Hypothesized  structure  is  accepted  as  appropriate 

•  Transfer  functions: 

-  Range  model  for  enemy  second  echelon  force 

information  data 

-  Range  model  for  vehicle  data 

-  Relative-weight  averaging  model  with  initial 

impression  for  emitter  data 
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In  summary,  analyses  of  the  TIQ  data  indicate  that: 

(1)  the  hypothesized  factors  all  affect  target  identification 
judgments  and  therefore  should  be  retained  in  the  structure; 

(2)  this  portion  of  the  immediate-targeting  structure  should 
consist  of  the  three  originally  hypothesized  causal  sets;  and 

(3)  the  three  transfer  functions  are  a  range  model  with  a  positive 
range  coefficient  for  Enemy  Second  Echelon  Force  Information,  a  range 
model  with  a  negative  range  coefficient  for  Vehicles,  and  a  relative- 
weight  averaging  model  with  an  initial  impression  for  Emitters.  The 
algebraic  formulations  of  these  models  was  presented  in  Slide  11. 


I 


FINAL  STRUCTURE  AND  STFs 

I  Battlefield  air  interdiction  I 
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This  slide  depicts  the  final  immediate-targeting  structure  with  the 
name  of  the  appropriate  transfer  function  for  each  causal  set. 

Once  the  STFs  have  been  determined,  immediate-targeting  outcomes 
can  be  predicted  for  any  specific  configuration  of  systems  that  support 
the  immediate-targeting  function.  This  would  be  done  in  the  following 
manner.  The  capabilities  of  the  support  systems  (such  as 
communications,  intelligence,  information  processing,  the  work  facility 
and  its  equipment,  reporting  systems,  etc.)  of  a  configuration  of 
interest  would  first  be  described  in  terms  of  the  factors  from  the  final 
representation.  For  example,  the  intelligence  system,  which  collects 
and  processes  raw  data  on  enemy  emitters  and  communicates  it  to  the 
command  and  control  system,  would  be  described  by  its  accuracy  in 
locating  emitters,  how  much  of  the  BAI  area  it  observes  (Coverage),  and 
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how  long  it  takes  to  process  the  data  and  report  it  to  the  command  and 
control  system  (Currency).  These  descriptions  would  reflect 
technological  capabilities  and  operational  concepts  and  procedures. 

If  the  description  is  a  physical  measure,  then  the  physical  measure 
would  be  entered  into  the  appropriate  utility  curve  (Slides  16  and  20) 
to  determine  the  appropriate  subjective  value  needed  for  its  STF.  If 
the  factor  is  described  verbally  (such  as  the  processing  and  emitter 
location  factors),  then  psychophysical  functions  (utility  curves)  do  not 
exist  and  the  description  must  be  one  of  those  used  in  the  development 
of  the  STF,  so  that  its  scale  value  is  known  (see  Appendixes  C  and 
D). [13] 

Once  the  subjective  scale  values  are  determined,  all  outcomes  are 
determined  by  computing  the  STFs  from  the  bottom  to  the  top  of  the 
hierarchical  structure.  A  description  of  such  a  computation  is 
presented  in  Appendix  E.  (In  practice,  the  STFs  would  be  incorporated 
into  a  computer  program  for  prediction  purposes.) 

The  next  seven  slides  illustrate  how  outcomes  from  different 
"notional"  system  configurations  can  be  determined  once  the  structure 
and  the  STFs  are  known.  These  outcomes  could  be  used  both  to  indicate 
relative  value  of  different  system  configurations  and  to  provide  input 
to  utility  models  used  in  the  theater  engagement  mission  area. 


[13]  If  the  description  is  new,  a  small  pilot  study  that  manipu¬ 
lates  only  the  factors  in  its  related  set  could  be  performed.  The  pur¬ 
pose  of  the  study  would  be  to  scale  the  new  descriptor,  but  it  would 
also  serve  as  a  validation  study  for  the  existing  STF. 
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To  assess  a  PDP  for  upgrading  sensor  data  processing  (such  as  a 
BETA  system,  for  example),  factor  levels  achieved  by  the  improved  system 
configuration  would  be  determined  and  the  outcome  predicted.  On  this 
slide,  a  notional  set  of  factor  levels  resulting  from  such  an  upgrade  is 
shown.  With  this  configuration,  the  percent  of  important  targets  in  the 
enemy  second  echelon  area  that  could  b„  identified  in  a  timely  manner  is 
predicted  to  be  40%-- an  improvement  of  7%  over  the  baseline 
configuration. 


50 


CAJL^Joc  ^  el***j 

All  wx,  loc 

Clr  wx,  loc  &  class 

Clr  wx,  loc 


observed 

Available  for  C' 

Accuracy. 

%  observed 

Available  for  C- 

90— 80% 

processing  in: 

10  m 

5o%  ass 

processing  in 

60% 

[5  min  J 

100  m 

OUT*  6Q% 

|5  mini 

30% 

1 5  min 

1000  m 

30% 

1  5  min 

10% 

30  min 

10% 

30  min 

1  hour 

1  hour 

Slide  25 


Now,  if  improvements  in  reconnaissance  and  surveillance  systems  are 
also  added,  they  may  bring  the  factor  levels  to  the  levels  shown  here. 
The  levels  of  80%  for  vehicle  coverage  and  50  meters  for  emitter 
accuracy  were  selected  to  demonstrate  that  any  values  (not  just  the 
values  contained  in  the  questionnaires)  can  be  used  for  factor  levels 
having  physical  measures.  For  this  configuration  the  predicted  outcome 
is  68%--a  28%  increase  over  the  notional  "BETA-like"-only  configuration 
and  a  35%  increase  over  the  baseline. 

We  next  consider  the  effect  these  configurations  would  have  on  the 


overall  measure  of  immediate-targeting  effectiveness. 
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APPLICATIONS  OF  THE  STFs  (Cent.) 
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This  slide  shows  the  range  of  immediate-targeting  outcomes,  given 
the  highest  factor  levels  and  the  lowest  factor  levels.  Recall  that  the 
outcomes  are  in  terms  of  the  percent  of  effective  force  application 
opportunities  that  could  be  exploited. 
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APPLICATIONS  OF  THE  STFs  (Com.) 
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In  this  example,  we  have  depicted  a  notional  baseline  configuration 
supporting  the  immediate-targeting  function  and  determined  outcomes  for 
both  the  baseline  and  enhanced  configuration  examples  previously  used 
for  enemy  second  echelon  force  information.  The  outcome  on  the  left, 
48%,  corresponds  to  33%  of  the  important  enemy  force  elements  being 
identified;  and  the  outcome  on  the  right,  52%,  corresponds  to  68%  of  the 
important  enemy  force  elements  being  identified. 

Hence,  in  this  configuration,  an  increase  of  over  100%  (33%  to  68%) 
in  the  percentage  of  important  enemy  force  elements  identified  is  judged 
to  produce  only  a  4%  increase  in  the  immediate-targeting  outcome 
measure.  This  may  indicate  the  perception  of  a  target-rich  (in  relation 
to  friendly  resources)  environment  in  the  conflict  scenario.  It  could 
also  indicate  that  other  factors  are  constraining  the  utilization  of  the 
additional  information.  Other  factors  can  be  easily  investigated  with 
this  structure. 
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APPLICATIONS  OF  THE  STFs  (Com.) 
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If  we  consider  an  upgrade  to  the  facilities,  including  better 
communications  capability,  which  would  achieve  the  notional  factor 
levels  shown  on  this  slide,  we  see  an  improvement  from  this  upgrade  that 
exceeds  the  improvement  attained  by  the  enhanced  enemy  information 
system  configuration  within  the  immediate-targeting  baseline  shown  on 
the  previous  slide.  In  the  baseline  (Slide  27),  the  enhanced  enemy 
information  configuration  yielding  68%  of  the  important  force  elements 
identified  achieved  an  immediate-targeting  outcome  of  52%  opportunities 
exploited.  On  the  other  hand,  using  the  enemy  information  configuration 
yielding  33%  of  the  important  force  elements  identified,  the  facility 
upgrade  alone  achieved  a  56%  immediate-targeting  outcome.  Whether  this 
difference  (52%  versus  56%)  is  considered  significant  is  up  to  the 
decisionmakers,  but  it  exemplifies  the  type  of  tradeoff  that  can  be 
disclosed  by  the  STFs  and  structure. 
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APPLICATIONS  OF  THE  STFs  (Cont.) 
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A  final  example  considers  only  a  notional  addition  to  airborne 
radar  and  communication  capability.  If  the  factor  levels  achieved  with 
such  an  upgrade  were  as  shown,  the  immediate-targeting  outcomes  would  be 
59%  and  65%.  Hence,  the  notional  outcomes  from  upgrading  facilties 
(previous  example)  or  adding  airborne  capabilities  are  judged  here  to  be 
about  the  same  in  terms  of  the  percent  of  effective  force  application 
opportunities  exploited  by  the  immediate-targeting  function. 

In  summary,  the  subjective  transfer  function  approach  provides  the 
framework  to  investigate  the  outcomes  of  individual  or  multiple  effects 
of  program  decision  packages  (PDPs)  throughout  the  structure,  and  hence 
permits  the  comparison  of  any  system  configurations  that  can  be 
described  by  the  factors. 
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STF FEATURES 


•  Contextual  causality 

-  Input  factors 

-  Input  factor  levels 

-  Outcomes 

•  Empirical  tests  of  STFs, 

factors,  and  structure 
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The  major  features  of  the  STF  approach  to  complex  system  analysis 
are  summarized  on  this  slide. 

1.  The  structure  depicts  a  context  of  sets  of  causal  hypotheses. 
The  sets  are  composed  of  factor  inputs  that  are  hypotheszied  to  impact 
on  outcomes.  Input  factors  are  defined  along  dimensions  of  interest 
(e.g.,  those  that  correspond  to  PDP  descriptions).  Outcomes  are  also 
operationally  defined  along  dimensions  of  interest  (e.g.,  those  that 
PDPs  are  supposed  to  affect). 

2.  The  structure  and  the  transfer  functions  that  specify  the  links 
among  the  sets  of  causal  hypotheses  are  empirically  verified. 

Appropriate  transfer  functions  are  used  to  predict  outcomes  throughout 
the  structure  and  at  the  top  of  the  structure. 
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SUMMARY 

The  STF  approach  develops  hypotheses  about 

•  Complex  system  factors 

•  Complex  system  structure 

•  Functional  links  among  causal  sets 

The  STF  approach  validates 

•  Hypothesized  factors 

•  Hypothesized  structures 

•  Functional  links  (STFs) 

The  STF  approach  is  useful  for 

•  Assessing  POPs 

•  Providing  valid  input  measures  to  other  models 
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The  STF  approach  provides  a  framework  for  capturing  the  experts' 
judgment  process  and  validating  the  experts'  measures  of  the  outcomes 
and  input  factors.  This  demonstration  study  has  illustrated  that  the 
STF  approach  is  an  advantageous  procedure  for  evaluating  complex 
systems --hypothesized  factors,  structures,  and  functional  links. 
Specifically,  we  have  demonstrated  the  usefulness  of  the  approach  for 
evaluating  perceived  effects  of  PDPs,  and  for  providing  AFWMAA  with 
valid  and  useful  subjective  measures  needed  for  computational  utility 
models . 
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Appendix  A 


METHODS  AND  PROCEDURES  FOR  THE 
IMMEDIATE  TARGETING  QUESTIONNAIRE 

Respondents  judged  questionnaire  items  that  described  factors  that 
affect  the  immediate-targeting  function  performed  within  the  tactical 
air  control  system.  Judgments  were  percentage  of  the  effective  force 
application  opportunities  that  could  be  exploited  by  the  immediate- 
targeting  function,  given  the  information  described  in  the  item. 

Independent  Variables 

The  definitions  and  selected  levels  of  the  factors  affecting 
immediate  targeting  are  described  below. 

Enemy  Information.  This  variable  reflects  the  amount  of  the  second 
echelon  force  elements  which  are  important  to  the  course  of  battle  that 
are  identified  in  time  for  use  in  performing  the  immediate-targeting 
function.  Four  levels  were  considered  as  follows: 

10%  of  the  important  2nd  echelon  force  elements  are  timely  identified 

30%  of  the  important  2nd  echelon  force  elements  are  timely  identified 

60%  of  the  important  2nd  echelon  force  elements  are  timely  identified 

90%  of  the  important  2nd  echelon  force  elements  are  timely  identified 

Facility  Operability .  This  factor  reflects  the  ability  of  the 
tactical  air  control  facility(ies)  to  support  the  immediate-targeting 
function.  It  would  include  consideration  of  survivability,  capacity, 
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and  endurance.  It  is  described  in  terms  of  the  percentage  of  all  the 
necessary  immediate-targeting  activities  that  the  facilities  could 
perform  if  all  effective  force  application  opportunities  were  to  be 
exploited.  Four  levels  were  considered: 

Facilities  can  support  10%  of  the  necessary  immediate-targeting 
activities . 

Facilities  can  support  30%  of  the  necessary  immediate-targeting 
activities . 

Facilities  can  support  60%  of  the  necessary  immediate-targeting 
activities . 

Facilities  can  support  90%  of  the  necessary  immediate-targeting 
activities . 

Dissemination  of  Tasking  Orders .  This  factor  reflects  the  ability 
to  correctly  communicate  tasking  orders  to  the  tactical  air  forces,  on 
the  ground  and  in  the  air,  that  are  designated  and/or  available  for 
second  echelon  interdiction  missions.  Four  levels  were  considered: 

Tasking  can  be  communicated  correctly  to  10%  of  the  forces  in  time. 

Tasking  can  be  communicated  correctly  to  30%  of  the  forces  in  time. 

Tasking  can  be  communicated  correctly  to  60%  of  the  forces  in  time. 

Tasking  can  be  communicated  correctly  to  90%  of  the  forces  in  time. 

Execution  Status  Information.  This  is  a  composite  factor  composed 
of  information  about  the  designated  alert  forces,  the  tactical  air 
forces  that  are  airborne,  and  the  weather.  Each  of  these  variables  is 
described  separately. 


St 
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Alert  Force  Status  Information:  This  factor  reflects  the  ability 
to  access  information  about  the  status  of  the  designated  alert  forces  in 
time  to  be  used  by  the  immediate-targeting  function.  Four  levels  were 
cons idered : 

Timely  access  to  the  status  of  10*.  of  the  designated  alert  forces. 

Timely  access  to  the  status  of  30°,  of  the  designated  alert  forces. 

Timely  access  to  the  status  of  60%  of  the  designated  alert  forces. 

Timely  access  to  the  status  of  90%  of  the  designated  alert  forces. 

Airborne  Force  Status  Information:  This  factor  reflects  the 
ability  to  access  information  about  the  status  of  tactical  air  forces 
that  are  airborne  in  time  to  be  used  by  the  immediate-targeting 
function.  Four  levels  were  considered: 

Timely  access  to  the  status  of  10%  of  the  airborne  forces. 

Timely  access  to  the  status  of  30%  of  the  airborne  forces. 

Timely  access  to  the  status  of  60%  of  the  airborne  forces. 

Timely  access  to  the  status  of  90%  of  the  airborne  forces. 

Weather  Data:  This  factor  reflects  the  currency  of  reliable 
weather  data  in  the  second  echelon  area  and  at  the  tactical  air  bases. 
Four  levels  were  considered: 

Weather  data  are  12  hrs .  old. 
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1 

0 

Weather  data  are  3  hrs .  old. 

Weather  data  are  1  hr.  old. 

Weather  data  are  15  min.  old. 


Stimuli  and  Design 

There  were  192  items  presented  for  judgment  in  the  ITQ.  These  were 

constructed  from  single  factors  or  factor  combinations.  To  include  all 

possible  combinations  of  four  levels  of  each  of  the  6  variables  and  all 

possible  subsets  would  require  15,624  items  (6x4  + 

2  3  4  5  6 

15  x  4  +20x4  +15x4  +6x4  +4).  Therefore,  a  subset  of 

i  possible  designs  were  chosen  that  met  the  following  criteria: 

collectively  they  would  (a)  include  the  complete  range  of  set-size  (from 

1  to  6  factors),  (b)  allow  assessment  of  all  interactions  of  interest, 

(c)  constrain  parameter  estimates,  and  (d)  permit  tests  among  the  models 

under  investigation.  The  192  items  were  generated  from  the  designs 

described  below. 

1.  Single  factor  designs .  For  24  items,  respondents  based  their 
judgments  on  only  one  of  the  six  variables;  each  of  the  six  variables 
had  four  levels. 

2.  Execution  status  information  designs .  Fifty-six  items 
contained  combinations  of  the  factors  reflecting  execution  status 
information.  Forty-eight  items,  in  which  two  pieces  of  execution  status 
information  were  available,  were  constructed  from  three  4x4  designs-- 


Alert  Force  Status  x  Airborne  Force  Status,  Alert  Force  Status  x  Weather 


Data,  and  Airborne  Force  Status  x  Weather  Data.  Eight  three-factor 
items  were  generated  from  a  2x2x2,  Alert  x  Airborne  x  Weather,  design; 
the  top  (best)  and  bottom  (worst)  levels  of  each  of  the  three  execution 
status  information  variables  were  used  in  this  design. 

3.  Execution  status  information  in  combination  with  the  other 
three  factors  made  up  another  80  items.  Sixteen  items  were  generated 
from  a  4x4,  Execution  Status  Information  x  Dissemination  of  Tasking 
Orders,  design,  where  the  four  levels  of  Dissemination  were  crossed  with 
four  sets  of  Execution  Status  Information- -the  combined  top  levels  from 
each  of  three  execution  status  variables,  the  combined  bottom  levels 
from  each  of  the  three  execution  status  variables,  and  two  other 
combinations  made  up  of  the  top  levels  of  Alert  and  Weather  with  the 
bottom  level  of  Airborne,  and  the  top  level  of  Airborne  with  the  bottom 
levels  of  Alert  and  Weather.  Sixteen  items  were  generated  from  a  4x4, 
Execution  Status  Information  x  Facility  Operability,  design,  where  the 
four  levels  of  Facility  Operability  were  crossed  with  the  same  four  sets 
of  Execution  Status  Information  used  in  the  Execution  Status  Information 
x  Dissemination  design.  Thirty-two  items  were  generated  from  an  8x4, 
Execution  Status  Information  x  Enemy  Information,  design,  where  the 
eight  Execution  Status  Information  sets  were  constructed  from  a  2x2x2, 
Alert  x  Airborne  x  Weather,  design;  the  top  and  bottom  levels  of  each  of 
the  three  execution  status  information  variables  were  used.  Sixteen 
items  were  generated  from  a  2x2x2x2,  Execution  Status  Information  x 
Dissemination  of  Tasking  Orders  x  Facility  Operability  x  Enemy 
Information,  design;  the  two  sets  of  Execution  Status  Information 
consisted  of  one  set  of  top  levels  and  one  set  of  bottom  levels  of  the 
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three  execution  status  variables;  the  two  levels  of  Dissemination, 
Facility,  and  Enemy  Information  were  the  top  and  bottom  levels  for  each. 

4.  Dissemination  of  Tasking  Orders  by  Facility  Operabi 1 ity . 

Sixteen  items  were  constructed  from  a  4x4,  Dissemination  x  Facility 
Operability,  design. 

5.  Facility  Operability  by  Enemy  Information .  Sixteen  items  were 
constructed  from  a  4x4,  Facility  x  Enemy  Information,  design. 


Item  Examples 

The  following  are  examples  of  items  consisting  of  different  numbers 

of  pieces  of  information.  An  example  of  an  item  containing  only  one 

piece  of  information  that  describes  Dissemination  capability  would  be: 

Tasking  can  be  correctly  communicated  to  90?o  of  the  forces  in  time. 

An  example  of  an  item  containing  three  pieces  of  information  would  be: 

Timely  access  to  the  status  of  90%  of  the  designated  alert  forces. 

Timely  access  to  the  status  of  90%  of  the  airborne  forces. 

Weather  data  are  15  min.  old. 

An  example  of  an  items  containing  information  about  all  factors  would  be: 

ENEMY  INFO:  90%  of  important  2nd  echelon  force  elements  are  timely 
identified . 

EXECUTION  STATUS  INFO:  Have  timely  access  to  the  status  of 
10%  of  the  designated  alert  forces  and  10?i  of  the 
airborne  forces;  weather  data  are  15  min  old. 

DISSEM:  Tasking  can  be  correctly  communicated  to  10%  of  the  forces  in  time 
FACILITIES  can  support  90%  of  the  necessary  immediate-targeting  activities 


Procedure  and  Task 

The  192  items  were  printed  in  random  order  on  21  pages  forming  a 


questionnaire  booklet.  Page  orders  were  randomized  for  each  respondent. 
Each  booklet  also  contained  a  set  of  instructions  and  18  representative 
warm-up  items . 
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Before  beginning  the  questionnaire,  respondents  were  given  an 
informational  briefing  on  the  background  and  goals  of  the  research,  and 
a  descriptive  briefing  on  the  conflict  situation  to  be  used  as  a 
backdrop  in  formulating  their  judgments. 

The  respondent's  task  was  to  judge  the  percentage  of  the  effective 
force  application  opportunities  that  could  be  exploited  by  the 
immediate-targeting  function,  given  only  the  information  described  in  an 
item.  Respondents  worked  at  their  own  pace. 
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Appendix  B 

METHODS  AND  PROCEDURES  FOR  THE 
TARGET  IDENTIFICATION  QUESTIONNAIRE 

Respondents  judged  questionnaire  items  that  described 
characteristics  of  enemy  second  echelon  force  information  available  to 
the  tactical  air  control  system  for  making  immediate-targeting 
decisions.  They  estimated  percent  of  important  second  echelon  force 
elements  that  could  be  identified  in  a  timely  manner,  given  the 
described  characteristics. 

Independent  Variables 

The  characteristics  of  enemy  second  echelon  force  information 
relate  to  the  quality  of  information  on  enemy  vehicles  and  emitters,  and 
the  process  used  to  interpret  this  information.  Enemy  vehicle 
information  was  characterized  by  location  and  classification 
capabilities,  coverage  capabilities,  and  the  currency  of  the 
information.  Information  on  enemy  emitters  was  characterized  by  its 
location  accuracy,  coverage,  and  currency.  The  definitions  and  selected 
levels  of  these  characteristics  are  described  below. 

Enemy  Vehicle  Location  and  Classification .  This  characteristic 
reflects  the  ability  of  sensor  systems  to  locate  and  classify  enemy 
vehicles.  Four  levels  of  this  characteristic  were  considered: 

Can  locate  and  classify  vehicles  in  all  weather. 


Can  locate  (but  not  classify)  vehicles  in  all  weather. 
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Can  locate  and  classify  vehicles  in  clear  weather  only. 

Can  locate  (but  not  classify)  vehicles  in  clear  weather  only. 

Enemy  Vehicle  Coverage ■  This  characteristic  reflects  how  many  of 
the  enemy  vehicles  in  the  second  echelon  area  have  been  observed.  Four 
levels  of  this  characteristic  were  considered: 

10%  of  the  second  echelon  vehicles  are  observed. 

30%  of  the  second  echelon  vehicles  are  observed. 

60%  of  the  second  echelon  vehicles  are  observed. 

90%  of  the  second  echelon  vehicles  are  observed. 

Enemy  Vehicle  Currency .  This  characteristic  reflects  the  time 
interval  between  the  observation  of  vehicles  in  the  second  echelon  area 
and  the  data's  availability  for  processing  in  the  command  and  control 
system.  Four  levels  of  this  characteristic  were  considered: 

Vehicle  information  is  available  for  processing  1  hr.  after  observation. 

Vehicle  information  is  available  for  processing  30  min.  after  observation 

Vehicle  information  is  available  for  processing  15  min.  after  observation 

Vehicle  information  is  available  for  processing  5  min.  after  observation. 

Enemy  Emitter  Location  Accuracy .  This  characteristic  reflects  the 
accuracy  with  which  enemy  emitters  are  located  in  the  enemy  second 
echelon  area.  Three  levels  of  the  characteristic  were  considered: 


Emitters  are  located  within  a  1000m  accuracy. 

Emitters  are  located  within  a  100m  accuracy. 

Emitters  are  located  within  a  10m  accuracy. 

Enemy  Emitter  Coverage.  This  characteristic  reflects  how  many  of 
the  enemy  emitters  in  the  second  echelon  area  have  been  observed.  Four 
levels  of  this  characteristic  were  considered: 

10%  of  the  second  echelon  emitters  are  observed. 

30%  of  the  second  echelon  emitters  are  observed. 

60%  of  the  second  echelon  emitters  are  observed. 

90%  of  the  second  echelon  emitters  are  observed. 

Enemy  Emitter  Currency.  This  characteristic  reflects  the  time 
interval  between  the  observation  of  emitters  in  the  second  echelon  area 
and  the  data's  availability  for  processing  in  the  command  and  control 
system.  Four  levels  of  this  ccharacteristic  were  considered: 

Emitter  information  is  available  for  processing  1  hr.  after  observation. 

Emitter  information  is  available  for  processing  30  min.  after  observation 

Emitter  information  is  available  for  processing  15  min.  after  observation 

Emitter  information  is  available  for  processing  5  min.  after  observation. 

Processing.  This  characteristic  is  the  means  by  which  enemy 
vehicle  and  emitter  information  is  interpreted.  We  considered  four 
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levels,  which  varied  by  extent  of  computerized  support.  One  had  no 
automation:  humans  interpret  the  information  by  sorting  through  hard 
copy,  textual  materials  (e.g.,  formatted  reports).  The  other  three 
levels  included  automation  and  computer  output  devices.  The  human  uses 
the  computer  to  organize,  select,  search,  and  display  the  information  to 
facilitate  interpretation.  Graphic  displays  present  pictures  made  up  of 
symbols  and  line  drawings,  e.g.,  a  map  with  military  units,  towns, 
roads,  and  terrain  features.  In  the  fully  computerized  case,  the 
computer  organizes  and  interprets  the  information.  The  four  levels  were 
identified  in  the  questionnaire  as  follows: 

Human  sorts  hard  copy,  textual  information;  human  interpretation. 

Human  uses  computer  to  sort  textual  information;  human  interpretation. 

Human  uses  computer  to  graphically  display  information;  human 
interpretation. 

Fully  computerized  interpretation. 

Stimuli  and  Design 

There  were  203  items  presented  for  judgment  in  the  TIQ.  These  were 

constructed  from  single  characteristics  or  combinations  of 

characteristics.  To  include  all  possible  combinations  of  the  levels  of 

each  of  the  seven  variables  described  above  would  require  62,499  items 

2 

(6  x  4)  +  (1  x  3)  =  27  single-characteristic  items;  15  x  4  +6x4x3=  312 

3  2 

two-characteristic  items;  20  x  4  +15x4  x3=  2,000  three-characteristic 

4  3  5 

items;  15  x  4  +20x4  x3=  7,680  four-characteristic  items;  6x4 


*  T 
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4  6  5 

+  15  x  4  x  3  =  17,664  five-characteristic  items;  1x4  +6x4  x3 

6 

=  22,528  six-characteristic  items;  4  x  3  =  12,288  seven-characteristic  items) 
Therefore,  we  chose  a  subset  of  possible  designs  that  would  include  the 
complete  range  of  set-size  (from  1  to  7  characteristics),  would  allow 
assessment  of  all  interactions  of  interest,  would  constrain  parameter 
estimates,  and  would  permit  tests  among  the  models  under  investigation. 

The  designs  that  generated  the  203  items  are  described  below. 

1.  Enemy  vehicle  information  designs .  Sixty-eight  items  contained 
one  or  more  pieces  of  enemy  vehicle  information.  For  12  items, 
respondents  based  their  judgments  on  only  one  characteristic  of  enemy 
vehicle  information;  each  of  the  three  variables  had  four  levels. 

Forty-eight  items,  in  which  two  pieces  of  enemy  vehicle  information  were 
available,  were  constructed  from  three  4x4  designs --Location  x  Coverage, 
Coverage  x  Currency,  and  Location  x  Currency.  Eight  three - 
characteristic  items  were  generated  from  a  2x2x2,  Location  x  Coverage  x 
Currency  design;  the  top  (best)  and  bottom  (worst)  levels  of  each  of  the 
three  vehicle  information  variables  were  used  in  this  design. 

2.  Enemy  emitter  information  designs .  Fifty-nine  items  contained 
one  or  more  pieces  of  enemy  emitter  information.  For  11  items, 
respondents  based  their  judgments  on  only  one  characteristic  of  enemy 
emitter  information;  Coverage  and  Currency  each  had  four  levels  and 
Location  Accuracy  had  three  levels.  Forty  items,  in  which  two  pieces  of 
enemy  vehicle  information  were  available,  were  constructed  from  two  4x3 
designs --Location  x  Coverage  and  Location  x  Currency--and  one  4x4 
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design- -Coverage  x  Currency.  Eight  three-characteristic  items  were 
generated  from  a  2x2x2,  Location  x  Coverage  x  Currency  design;  the  top 
and  bottom  levels  of  each  of  the  three  emitter  information  variables 
were  used  in  this  design. 

3.  Vehicle  by  emitter  design.  Sixteen  items  were  constructed  from 
a  4x4,  Vehicle  x  Emitter,  design.  The  four  levels  of  Vehicle 
information  were  made  up  of  the  combined  top  levels  from  each  of  the 
three  vehicle  variables,  the  combined  bottom  levels  from  each  of  the 
three  vehicle  variables,  and  two  other  combinations--the  top  levels  of 
Coverage  and  Currency  with  the  bottom  level  of  Location  and  the  top 
level  of  Currency  with  the  bottom  levels  of  Location  and  Coverage.  The 
four  levels  of  Emitter  information  were  made  up  of  the  combined  top 
levels  from  each  of  the  three  emitter  variables,  the  combined  bottom 
levels  from  each  of  the  three  emitter  variables,  and  two  other 
combinations --the  top  levels  of  Location  and  Coverage  with  the  bottom 
level  of  Currency  and  the  top  level  of  Coverage  with  the  bottom  levels 
of  Location  and  Currency. 

4.  Processing.  This  factor,  alone  and  in  combination  with  vehicle 
or  emitter  information,  made  up  another  52  items.  For  four  items, 
respondents  based  their  judgments  on  only  the  Processing  variable,  which 
had  four  levels.  Twenty-four  items  were  constructed  from  an  8x3, 

Emitter  x  Processing,  design,  where  the  eight  Emitter  Information  sets 
were  constructed  from  a  2x2x2,  Location  x  Coverage  x  Currency,  factorial 
design,  using  the  top  and  bottom  levels  of  each  of  these  three  emitter 
information  variables  and  three  levels  of  the  processing  variable--human 
sorts  hard  copy  . . . ,  human  uses  computer  to  sort  textual  information 
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. . .  ,  and  fully  computerized  interpretation  (these  are  completely  stated 
in  the  section  titled  "independent  Variables") .  Twenty-four  items  were 
constructed  from  a  2x12,  Processing  x  Vehicle,  design,  where  the  12 
Vehicle  Information  sets  were  constructed  from  a  2x2x3,  Location  x 
Coverage  x  Currency,  factorial  design,  using  the  top  and  bottom  levels 
of  Location  and  Coverage  and  three  levels  of  Currency--the  two  top 
levels  and  the  bottom  level  (5  min.,  15  min.,  and  1  hr.);  the  two  levels 
of  Processing  used  were  the  top  and  bottom  levels. 

5.  Vehicle  by  processing  by  emitter  design.  Eight  items  were 
generated  from  a  2x2x2,  Vehicle  x  Processing  x  Emitter,  design.  The  top 
and  bottom  levels  of  Processing  were  used.  The  two  sets  of  Vehicle 
Information  consisted  of  one  set  of  top  levels  and  one  set  of  bottom 
levels  of  the  three  Vehicle  Information  variables.  The  two  sets  of 
Emitter  Information  were  constructed  in  the  same  manner. 

Item  Examples 

The  following  are  examples  of  items  consisting  of  different  numbers 
of  characteristics.  An  example  of  an  item  containing  only  one 
characteristic  would  be: 

90 %  of  the  second  echelon  Emitters  are  observed. 

An  example  of  an  item  containing  three  characteristics  would  be: 

Vehicle  info  is  available  for  processing  5  min.  after  observation. 

Can  locate  and  classify  vehicles  in  all  weather. 

90%  of  the  second  echelon  Vehicles  are  observed. 

An  example  of  an  item  consisting  of  all  seven  characteristics  would  be: 

Locate  and  classify,  in  all  weather,  90%  of  the  second  echelon  vehicles 
vehicle  info  is  available  for  processing  5  min  after  observation. 

Locate  (within  a  10m  accuracy)  90%  of  the  second  echelon  emitters; 
emitter  info  is  available  for  processing  5  min  after  observation. 
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Human  sorts  hard  copy,  textual  info;  human  interpretation. 

Procedure  and  Task 

The  203  items  were  printed  in  random  order  on  22  pages  forming  a 
questionnaire  booklet.  Page  orders  were  randomized  for  each  respondent. 
Each  booklet  also  contained  a  set  of  instructions  and  18  representative 
warm-up  trials. 

As  in  the  ITQ,  before  beginning  the  questionnaire,  respondents  were 
given  an  informational  briefing  on  the  background  and  goals  of  the 
research,  and  a  descriptive  briefing  on  the  conflict  situation  to  be 
used  as  a  backdrop  in  formulating  their  judgments. 

The  respondent's  task  was  to  judge  the  percentage  of  the  important 
second  echelon  force  elements  that  could  be  identified  in  a  timely 
manner,  considering  the  context  of  immediate  targeting  in  the  second 
echelon  area  and  given  only  the  characteristic(s)  described  in  an  item. 
Respondents  worked  at  their  own  pace. 


72 

Appendix  C 

WEIGHTS  AND  SCALE  VALUES  FOR  THE 
IMMEDIATE  TARGETING  QUESTIONNAIRE 


SCALE  VALUES  (s) 


Factor 

Level 

1 

2 

3 

4 

Enemy  Information 

-1.092 

4.309 

8.715 

14.255 

Facility  Operability 

-0.634 

3.009 

7.145 

9.519 

Alert  Force  Status  Information 

1.824 

5.123 

7.093 

9.192 

Airborne  Tactical  Force  Status 

2.412 

5.357 

9.755 

16.528 

Weather  Currency 

0.532 

3.124 

6.141 

10.935 

Dissemination  of  Tasking  Orders 

-0.464 

3.496 

7.973 

10.806 

Initial  Impression  (sQ)  =  2.303 


WEIGHTS  (w) 


Factors  for  Immediate  Targeting  Node 


Enemy  Information  0.654 
Facility  Operability  0.865 
Alert  Force  Status  Information  0.614 
Airborne  Tactical  Force  Status  0.503 
Weather  Currency  0.328 
Dissemination  of  Tasking  Orders  1.000 

Range  coeffficient  (u)  =  -0.079 


Initial  Impression  (wQ) 


1.554 
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WEIGHTS  AND  SCALE  VALUES  FOR  THK 
TARGET  IDENTIFICATION  QUESTIONNAIRE 


Factor 

Level 

1 

2 

3 

4 

Enemy  Vehicle  Location 

3.54 

7.03 

10.12 

17.44 

Enemy  Vehicle  Coverage 

0.28 

2.97 

8.04 

14.32 

Enemy  Vehicle  Currency 

1.43 

3.94 

8.04 

11.99 

Enemy  Emitter  Location 

2.24 

6.88 

12.16 

Enemy  Emitter  Coverage 

0.58 

2.37 

5.08 

8.33 

Enemy  Emitter  Currency 

-3.56 

3.95 

8.97 

18.74 

Processing 

4.02 

3.80 

4.51 

3.75 

Initial  Impression  (sq) 

=  1.096 

WEIGHTS  (w) 


Factors  for  Vehicle  and  Emitter  Nodes 


Vehicle  Factors 

Emitter  Factors 

Enemy  Vehicle  Location 

0.176 

Enemy  Emitter  Location 

0.235 

Enemy  Vehicle  Coverage 

0.434 

Enemy  Emitter  Coverage 

1.187 

Enemy  Vehicle  Currency 

0.240 

Enemy  Emitter  Currency 

0.115 

Range  coefficient  -  Vehicles  (u)  =  -0.048 


Factors  for  Enemy  Second  Echelon  Force  Information  Node 


Vehicles 

25 

.950 

Emitters 

24, 

.552 

Processing 

1 

.000 

Range  coefficient  -  Enemy  second  echelon  force  information  Qv)  =  0.320 

Initial  Impression  (wQ)  =  0.972 


Appendix  E 

DESCRIPTION  OF  HOW  TO  USE  STFs 


The  predictions  are  arrived  at  by  substituting  the  appropriate 
numerical  values  from  Appendix  D  (weights  and  scale  values  for  TIQ)  into 
the  appropriate  model  formulas  given  on  Slide  11.  For  example,  the 
prediction  of  33%  is  arrived  at  by  the  following  procedure: 

1.  The  weights  derived  for  the  vehicle  factors  and  the  scale 
values  derived  for  the  selected  set  of  vehicle  factor  levels  are  given 
in  Appendix  D,  together  with  the  other  parameter  estimates  for  the  range 
model.  These  numerical  values  are  substituted  into  the  range  model  (Eq. 
2  on  the  right-hand  panel  of  Slide  11).  The  output  value  of  2.11  is 
obtained  for  the  selected  vehicle  combination. 

2.  The  relative  weight  model  (Eq.  3  on  the  left-hand  panel  of 
Slide  11)  parameters  needed  for  the  emitter  factors  are  given  in  Appen¬ 
dix  D.  Substituting  these  values  into  the  model  produces  an  output 
value  of  3.31  for  the  selected  emitter  combination. 

3.  The  output  values  for  the  vehicle  node  and  the  emitter  node 
become  the  scale  values  (s)  for  those  factors  in  the  range  model  at  the 
enemy  second  echelon  force  information  node.  The  remaining  parameters 
needed  (weights  for  Vehicles,  Processing,  and  Emitters;  scale  value  for 
the  given  level  of  Processing;  initial  impression  weight  and  scale 
value;  and  the  range  coefficient)  are  given  in  Appendix  D.  The  range 
model  with  these  values  produces  a  predicted  response  of  3.3  (on  a  1  to 
9  response  scale)  which  corresponds  to  a  percent  response  of  33%. 


